Conversation
aron-cf
marked this pull request as ready for review
June 15, 2026 14:45
The loopback-suppression optimization in applyChanges advanced the local pushRev to currentRev after every upstream apply, on the theory that the apply's own rev bumps would otherwise get re-pushed. That theory was correct in isolation but the implementation was unsound: it moved our pushRev past entries the remote did not know we had shipped, while the remote's fetchRev (echoed back as appliedPushRev on every fetchChanges response) stayed where the last real push had put it. The cross-side invariant check in pullOnce then trips on the very next pull and the post-drain pull in the exec bracket swallows the error, leaving every subsequent container-side write invisible to the host until something reconciles the watermarks. Drop the local advance. The next pushOnce ships the apply's rev bumps, the receiver's alreadyApplied() check drops them as no-ops, and the container's fetchRev catches up to our pushRev in the same round trip. One extra push per upstream apply, bounded by the batch's coalesce output. The cross-side invariant stays intact. The two F1 tests on this behavior already covered the unsafe case (do not strand unpushed locals); they still pass. The two tests that pinned the optimization (one in apply.test.ts, one in wire.test.ts) are updated to assert the new contract.
reconcileWatermarks runs once per connect and resets local cursors to 0 when the remote is behind us. That handles the WebSocket-drop case but not the WebSocket-survives-wsd-restart case: wsd's store is process-lifetime, so a wsd respawn under the same WS leaves the DO holding cursors the container no longer knows about, and the next pull's cross-side invariant assertion throws. Move the recovery inline. When fetchChanges reports an appliedPushRev below our localPushRev, or a currentRev below our fetchRev, treat it as a real-time reconcile: cancel the in-flight stream, reset the divergent cursor to 0, and recurse once. The rev-0 baseline path re-ships incrementally and the receiver's alreadyApplied() check absorbs the work. A second divergence after the retry surfaces via the existing assertion, so a persistently broken remote still fails loudly. Combined with the prior pushRev-locality fix this closes the 'FUSE write invisible to DO readFile' bug observed on the deployed container example: the apply-side fix prevents the divergence from being introduced, and this inline recovery prevents any future divergence (mid-flight restart, harness shenanigans, ...) from wedging the same way.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Container-side filesystem writes were not making it back to the durable object after the first exec. A burst of
printf > /workspace/foo.txtcalls each succeeded inside the container but the host'sws.fs.readFilereturned ENOENT for every file after the first. Once a workspace fell into this state, subsequent execs could not pull the missing entries even though the bytes were inwsd's store.The root cause was a loopback-suppression optimization inside
applyChanges. After applying an upstream batch the apply path advanced the localpushRevto the post-applycurrentRev, moving it past entries the container had no way to know we had shipped. The container'sfetchRevstayed where the last real push put it. The cross-side invariant check insidepullOncethen tripped on the next pull, the exec bracket swallowed the error, andfetchRevcould no longer advance.The fix has two parts. Stop advancing
pushRevlocally on upstream apply: the receiver'salreadyAppliedcheck drops the redundant entries on the nextpushOnceand the container'sfetchRevcatches up in the same round trip. The cost is one extra push per upstream apply, bounded by the batch. MakepullOncerecover inline when a divergence does occur: cancel the stream, reset the divergent cursor to zero, retry once. A second divergence after the retry still surfaces via the existing assertion.Unit tests cover both halves. The dofs apply suite pins the new contract:
pushRevstays put on upstream apply, the next coalesce surfaces the apply's rev bumps, and the receiver's idempotence does the rest. The rpc suite pins the inline recovery: a one-shot lying remote triggers a reset and the retried call drains normally; a persistently lying remote degrades to a baseline re-sync rather than throwing.